Time-of-flight sensor with structured light illuminator

ABSTRACT

The present disclosure relates to systems and methods that provide information about a scene based on a time-of-flight (ToF) sensor and a structured light pattern. In an example embodiment, a sensor system could include at least one ToF sensor configured to receive light from a scene. The sensor system could also include at least one light source configured to emit a structured light pattern and a controller that carries out operations. The operations include causing the at least one light source to illuminate at least a portion of the scene with the structured light pattern and causing the at least one ToF sensor to provide information indicative of a depth map of the scene based on the structured light pattern.

BACKGROUND

Time-of-Flight (ToF) sensors typically provide low-resolution depthinformation about a scene, but can be subject to stray light “blooming”and/or provide inaccurate depth information when imaging highlyreflective or highly absorbing materials.

Structured light can include light emitted according to a desired orpredetermined illumination pattern and/or illumination schedule. Somelight sources may be configured to illuminate a scene with structuredlight.

SUMMARY

The present disclosure beneficially combines aspects of ToF sensors andstructured light to provide more accurate, higher-resolution depthinformation.

In a first aspect, a sensor system is provided. The sensor systemincludes at least one time-of-flight (ToF) sensor configured to receivelight from a scene. The sensor system also includes at least one lightsource configured to emit a structured light pattern. Furthermore, thesensor system includes a controller that carries out operations. Theoperations include causing the at least one light source to illuminateat least a portion of the scene with the structured light pattern. Theoperations also include causing the at least one ToF sensor to provideinformation indicative of a depth map of the scene based on thestructured light pattern.

In a second aspect, a system is provided. The system includes aplurality of sensor systems configured to be coupled to a vehicle. Eachsensor system includes at least one time-of-flight (ToF) sensor and atleast one imaging sensor. The at least one ToF sensor and the at leastone imaging sensor are configured to receive light from a scene. Eachsensor system also includes at least one light source configured to emita structured light pattern and a controller that carries out operations.The operations include causing the at least one light source toilluminate at least a portion of the scene with the structured lightpattern. The operations also include causing the at least one ToF sensorto provide information indicative of a depth map of the scene based onthe structured light pattern. The operations additionally includecausing the imaging sensor to provide information indicative of an imageof the scene based on the structured light pattern.

In a third aspect, a method is provided. The method includes causing atleast one light source to illuminate a scene with a structured lightpattern. The method additionally includes receiving, from atime-of-flight (ToF) sensor, information about the scene based on thestructured light pattern. The method also includes determining a depthmap of the scene based on the received information. The method yetfurther includes determining at least one inference about the scenebased on the depth map of the scene.

In a fourth aspect, a method is provided. The method includes providingprior information. The prior information includes three-dimensionalinformation of a scene. The method includes causing at least one lightsource to illuminate the scene with a structured light pattern. Themethod also includes causing the at least one ToF sensor to provide timeof flight information indicative of a depth map of the scene based onthe structured light pattern.

Other aspects, embodiments, and implementations will become apparent tothose of ordinary skill in the art by reading the following detaileddescription, with reference where appropriate to the accompanyingdrawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a system, according to an example embodiment.

FIG. 2 illustrates an operating scenario of a system, according toexample embodiments.

FIG. 3A illustrates a vehicle, according to an example embodiment.

FIG. 3B illustrates a sensor unit, according to an example embodiment.

FIG. 3C illustrates a light source, according to an example embodiment.

FIG. 4A illustrates a sensing scenario, according to an exampleembodiment.

FIG. 4B illustrates a sensing scenario, according to an exampleembodiment.

FIG. 4C illustrates various structured light patterns, according toexample embodiments.

FIG. 4D illustrates a structured light pattern, according to an exampleembodiment.

FIG. 5 illustrates a method, according to an example embodiment.

FIG. 6A illustrates a sensing scenario, according to an exampleembodiment.

FIG. 6B illustrates a sensing scenario, according to an exampleembodiment.

FIG. 7 illustrates a method, according to an example embodiment.

DETAILED DESCRIPTION

Example methods, devices, and systems are described herein. It should beunderstood that the words “example” and “exemplary” are used herein tomean “serving as an example, instance, or illustration.” Any embodimentor feature described herein as being an “example” or “exemplary” is notnecessarily to be construed as preferred or advantageous over otherembodiments or features. Other embodiments can be utilized, and otherchanges can be made, without departing from the scope of the subjectmatter presented herein.

Thus, the example embodiments described herein are not meant to belimiting. Aspects of the present disclosure, as generally describedherein, and illustrated in the figures, can be arranged, substituted,combined, separated, and designed in a wide variety of differentconfigurations, all of which are contemplated herein.

Further, unless context suggests otherwise, the features illustrated ineach of the figures may be used in combination with one another. Thus,the figures should be generally viewed as component aspects of one ormore overall embodiments, with the understanding that not allillustrated features are necessary for each embodiment.

I. Overview

Imaging sensors typically provide high quality, high-resolution,two-dimensional images of a scene, but do not typically provideindependent depth information. Time-of-Flight (ToF) sensors typicallyprovide low-resolution depth information about a scene, but can besubject to artifacts such as image blooming in the presence of highlyreflective objects or inaccurate depth measurements in the presence ofmixed scenes with reflective and absorptive objects. The presentdisclosure beneficially combines the desirable aspects of both types ofsensors to provide more accurate, higher-resolution depth information.

In some examples, blooming can occur when a given sensor pixel absorbsenough photons such that the number of photo-generated charge carriersexceeds a full well capacity (FWC). In such scenarios, upon reachingFWC, excess charge carriers can “overflow” into neighboring sensorpixels, producing a smearing or blurring effect, which may reduce imagequality and/or reduce confidence in depth information.

A hybrid imaging system could include: 1) at least one ToF sensor; 2) anoptional imaging sensor; 3) at least one light source for illuminatingthe scene with structured light using continuous wave (CW), pulsed, oraperiodic illumination; and 4) a controller, which may include acomputer, a processor, and/or a Deep Neural Net. The ToF sensor and theimaging sensor may be spatially registered to one another and mayutilize overlapping portions of the same optical path. For example, theToF sensor and the imaging sensor could be spatially registered to oneanother such that they have a similar (e.g., roughly identical) field ofview and their relative position and orientation could be known and/orfixed with respect to one other.

Each sensor unit of a plurality of sensor units of such a hybrid imagingsystem could be mounted on each side (or corner) of a vehicle.Respective sensor units could also be mounted in one or more spinningplatforms at various locations on the vehicle. In an example embodiment,each sensor unit may include a 180 degree field of view of a scenearound the vehicle. In some embodiments, sensor units could bepositioned on the vehicle so as to have partially overlapping fields ofview of the environment around the vehicle.

In an example embodiment, to avoid blooming or other depth informationartifacts, a plurality of ToF sensors could be associated with one ormore image sensors in a given sensor unit. The respective ToF sensorscould be spread out (e.g., spaced apart by 10 cm or more) so as toreduce the effects of blooming from specular reflections and otherbright light sources. In some embodiments, the ToF sensors could beoperated between 10-100 MHz, however other operating frequencies arecontemplated and possible. In some embodiments, the operating frequencyof the respective ToF sensor may be adjusted based on a desired maximumdepth sensing range. For instance, a ToF sensor could be operated at 20MHz for a desired depth sensing range (e.g., unambiguous range) ofapproximately 7.5 meters. In some embodiments, the ToF sensor could havea maximum desired depth sensing range of 100 meters or more.

In some embodiments, the ToF sensor could include CMOS or CCDphoto-sensitive elements (e.g., silicon PIN diodes). However, othertypes of ToF sensors and ToF sensor elements are contemplated. In somecases, the ToF sensor could be operated using various phase shift modes(e.g., a 2× or 4× phase shift).

In some embodiments, the imaging sensor could include an RGB imagingsensor, such as a megapixel-type camera sensor. The imaging sensor couldinclude a plurality of CMOS or CCD photo-sensitive elements.

In some examples, one or more light sources could be used to illuminatethe scene (or respective portions of the scene). In such scenarios, thelight sources could be modulated to provide a predetermined light pulse(or series of light pulses) that could be used in conjunction with theToF sensor to provide depth information. Additionally or alternatively,the series of light pulses (e.g., a pulse repetition rate, a pulseduration, and/or a duty cycle) could be selected so as to provide adesired exposure for the imaging sensor.

The one or more light sources could include a light strip that isdisposed along a portion of the vehicle. Additionally or alternatively,the one or more light sources could include a grid of light panels, eachsegment of which could individually provide different light pulses. Yetfurther, the one or more light sources could provide one or more lightbeams that can be moved in a point-wise and/or scanning fashion.

The one or more light sources could be operated in CW and/or in pulsed(e.g., sine wave, sawtooth, or square wave) operation mode. Withoutlimitation, the one or more light sources could include at least one of:a laser diode, a light-emitting diode, a plasma light source, a strobe,a solid-state laser, a fiber laser, or another type of light source. Theone or more light sources could be configured to emit light in theinfrared wavelength range (e.g., 850, 905, 940, and/or 1550 nanometers).In some embodiments, multiple illumination light wavelengths could beused to disambiguate between multiple light sources, etc. Additionallyor alternatively, the illumination wavelength may be adjusted based onan amount of ambient light in the environment and/or a time of day.

In another example embodiment, the one or more light sources could emita structured light pattern into the environment. The structured lightpattern could provide improved registration and/or resistance toblooming effects. As an example, the structured light pattern could beformed by transmitting light through a diffractive optic element. Inanother embodiment, a laser light pattern (e.g., random laser speckle orpredetermined laser light pattern) could be used to provide thestructured light pattern. In yet further embodiments, a deformable oradjustable reflective, diffractive, or refractive surface (e.g., amicromirror array) could be used to provide the structured light patternand/or to shift the pattern with respect to the scene.

Additionally or alternatively, the one or more light sources could beconfigured to emit one or more classes of structured light patterns. Forinstance, the classes of structured light patterns could include one ormore spatial classes, where some regions of a field of view areilluminated (or not illuminated) according to a predetermined spatiallight pattern. Other classes of structured light patterns could includetemporal classes, where various regions of a field of view areilluminated at different times according to a predetermined temporalillumination schedule. Yet other classes of structured light couldinclude spectral classes, where various regions of a field of view areilluminated with different wavelengths—or wavebands—of light accordingto a predetermined spectral illumination pattern. However, other ways toform a structured light pattern are possible and contemplated herein.

In some embodiments, the structured light pattern could be used todisambiguate spatial locations within a scene. For example, thestructured light pattern could include circular and/or oval-shaped light“spots”. Each spot could have a different shape or orientation (e.g.,rotation, spatial extent, radius of curvature, elongation, etc.) basedon, for example, an emission angle of light through the diffractiveoptic element or a spatial position in the scene with respect to thelight source. In some embodiments, a predetermined astigmatism of theoptical element could be utilized to disambiguate between light spots inthe structured light pattern.

The controller could be operable to combine outputs of the respectivesensors (e.g., using sensor fusion) and/or make inferences about thethree-dimensional scene around the vehicle. For example, the controllercould make inferences to provide a grayscale or color-intensity map ofthe vehicle's surroundings. The inferences may additionally oralternatively provide information about objects in the vehicle'senvironment. In an example embodiment, the object information could beprovided at a refresh rate of 60 or 120 Hz. However, other refresh ratesare possible and contemplated.

In an example embodiment, the system could include one or more deepneural networks. The deep neural networks(s) could be utilized toprovide the inferences based on training data and/or an operatingcontext of the vehicle. In some cases, the low-resolution depthinformation and the image information may be provided to the deep neuralnetwork. Subsequently, the deep neural network could make inferencesbased on the received information and/or provide output depth maps(e.g., point clouds) at a high-resolution.

In some embodiments, two or more of: the ToF sensor, the image sensor,the light source, and the controller could be coupled to the samesubstrate. That is, the system could include a monolithic chip orsubstrate so as to provide a smaller sensor package and/or provide otherperformance improvements.

II. Example Systems

FIG. 1 illustrates a system 100, according to an example embodiment. Thesystem 100 includes at least one Time-of-Flight (ToF) sensor 110, or ToFcamera. In an example embodiment, the at least one ToF sensor 110 couldinclude a plurality of complementary metal-oxide semiconductor (CMOS) orcharge-coupled device (CCD) photosensitive elements (e.g., silicon PINdiodes). Other types of photosensitive elements could be utilized by theToF sensor 110.

In some embodiments, the at least one ToF sensor 110 could be configuredto actively estimate distances to environmental features in itsrespective field of view based on the speed of light. For instance, theToF sensor 110 could measure the time-of-flight of a light signal (e.g.,a light pulse) upon traveling between a light source (e.g., light source130) and an object in the scene. Based on estimating the time-of-flightof light pulses from a plurality of locations within a scene, a rangeimage or depth map can be built up based on the ToF sensor's field ofview. While the distance resolution can be 1 centimeter or less, thelateral resolution can be low as compared to standard 2D imagingcameras.

In some embodiments, the ToF sensor 110 can obtain images at 120 Hz orfaster. Without limitation, the ToF sensor 110 could include arange-gated imager or a direct time-of-flight imager.

Optionally, the system 100 may also include at least one imaging sensor120. In an example embodiment, the imaging sensor 120 could include aplurality of photosensitive elements. In such a scenario, the pluralityof photosensitive elements could include at least one millionphotosensitive elements. The at least one ToF sensor 110 and the atleast one imaging sensor 120 are configured to receive light from ascene.

The system 100 also includes at least one light source 130. In anexample embodiment, the at least one light source 130 could include atleast one of: a laser diode, a light-emitting diode, a plasma lightsource, a strobe light, a solid-state laser, or a fiber laser. Othertypes of light sources are possible and contemplated in the presentdisclosure. The at least one light source 130 could include a lightstrip (e.g., disposed along a portion of a vehicle). Additionally oralternatively, the at least one light source 130 could include, forexample, a grid of light panels, each segment of which couldindividually provide different light pulses. Yet further, the at leastone light source 130 could provide one or more light beams that can bemoved in a point-wise and/or scanning fashion. The at least one lightsource 130 could be operated in a continuous wave (CW) mode and/or in apulsed (e.g., sine wave, sawtooth, or square wave) operation mode.

In an example embodiment, the at least one light source 130 could beconfigured to emit infrared light (e.g., 900-1600 nanometers). However,other wavelengths of light are possible and contemplated.

In some embodiments, the at least one light source 130 could beconfigured to emit light into the environment according to a desiredstructured light pattern. The structured light pattern could include,for example, aperiodic and/or inhomogeneous illumination of theenvironment by the at least one light source 130. For example, thedesired structured light pattern could include a checkerboard pattern, adot pattern, a stripe pattern, a speckle pattern, or anotherpredetermined light pattern. Additionally or alternatively, in someembodiments, pseudorandom light patterns are possible and contemplated.The desired structured light pattern could be defined by light pulses,or shots, emitted along a predetermined pointing angle and/or within apredetermined field of view. In some embodiments, the light pulses couldbe provided at different temporal and/or spatial/angular densities basedon the desired structured light pattern.

The at least one light source 130 and the ToF sensor 110 could betemporally synchronized. That is, a trigger signal to cause the lightsource 130 to emit light could also be provided to the ToF imager 110 asa temporal reference signal. As such, the ToF sensor 110 may haveinformation about a time of the actual onset of the light emitted fromthe light source 130. Additionally or alternatively, the ToF sensor 110could be calibrated based on a reference target at a known distance fromthe ToF sensor 110.

In scenarios with multiple light sources and/or multiple ToF imagers,the multiple light sources could utilize time multiplexing or othertypes of signal multiplexing (e.g., frequency or code multiplexing) soas to disambiguate time-of-flight information (light pulses) obtained bya given ToF imager from the various light sources.

In some embodiments, the at least one light source 130 could beconfigured to emit light into an environment along a plurality ofemission vectors toward various target locations so as to provide adesired resolution. In such scenarios, the at least one light source 130could be operable to emit light along the plurality of emission vectorssuch that the emitted light interacts with an external environment ofthe system 100.

In an example embodiment, the respective emission vectors could includean azimuthal angle and/or an elevation angle (and/or correspondingangular ranges) with respect to a heading or location of a vehicle(e.g., vehicle 300 as illustrated and described with reference to FIG.3A). In some embodiments, light emitted by the at least one light source130 could be directed along the respective emission vectors by adjustinga movable mount and/or a movable mirror.

For example, the at least one light source 130 could emit light toward amovable mirror. By adjusting an orientation of the movable mirror, theemission vector of the light could be controllably modified. It will beunderstood that many different physical and optical techniques may beused to direct light toward a given target location. All such physicaland optical techniques for adjusting an emission vector of light arecontemplated herein.

Optionally, the system 100 may include other sensors 140. The othersensors 140 may include a LIDAR sensor, a radar sensor, or other typesof sensors. For instance, system 100 could include a Global PositioningSystem (GPS), an Inertial Measurement Unit (IMU), a temperature sensor,a speed sensor, a camera, or a microphone. In such scenarios, any of theoperational scenarios and/or methods described herein could includereceiving information from the other sensors 140 and carrying out otheroperations or method steps based, at least in part, on the informationreceived from the other sensors 140.

In an example embodiment, at least two of: the at least one ToF sensor110, the imaging sensor 120, and the at least one light source 130 couldbe coupled to a common substrate. For example, the at least one ToFsensor 110, the imaging sensor 120, and the at least one light source130 could be coupled to a vehicle. In some embodiments, some or allelements of system 100 could provide at least a portion of the objectdetection and/or navigation capability of the vehicle. The vehicle couldbe a semi-autonomous or fully-autonomous vehicle (e.g., a self-drivingcar). For instance, system 100 could be incorporated into vehicle 300 asillustrated and described in reference to FIGS. 3A, 4A, 4B, 6A, and 6B.

In some embodiments, system 100 could be part of a vehicle controlsystem utilized to detect and potentially identify nearby vehicles, roadboundaries, weather conditions, traffic signs and signals, andpedestrians, among other features within the environment surrounding thevehicle 300. For example, a vehicle control system may use depth mapinformation to help determine control strategy for autonomous orsemi-autonomous navigation. In some embodiments, depth map informationmay assist the vehicle control system to avoid obstacles while alsoassisting with determining proper paths for navigation.

While some examples described herein include system 100 as beingincorporated into a vehicle, it will be understood that otherapplications are possible. For example, system 100 could include, or beincorporated into, a robotic system, an aerial vehicle, a smart homedevice, a smart infrastructure system, etc.

System 100 includes a controller 150. In some embodiments, thecontroller 150 could include an on-board vehicle computer, an externalcomputer, or a mobile computing platform, such as a smartphone, tabletdevice, personal computer, wearable device, etc. Additionally oralternatively, the controller 150 can include, or could be connected to,a remotely-located computer system, such as a cloud server network. Inan example embodiment, the controller 150 may be configured to carry outsome or all of the operations, method blocks, or steps described herein.Without limitation, the controller 150 could additionally oralternatively include at least one deep neural network, another type ofmachine learning system, and/or an artificial intelligence system.

The controller 150 may include one or more processors 152 and at leastone memory 154. The processor 152 may include, for instance, amicroprocessor, an application-specific integrated circuit (ASIC), or afield-programmable gate array (FPGA). Other types of processors,circuits, computers, or electronic devices configured to carry outsoftware instructions are contemplated herein.

The memory 154 may include a non-transitory computer-readable medium,such as, but not limited to, read-only memory (ROM), programmableread-only memory (PROM), erasable programmable read-only memory (EPROM),electrically erasable programmable read-only memory (EEPROM),non-volatile random-access memory (e.g., flash memory), a solid statedrive (SSD), a hard disk drive (HDD), a Compact Disc (CD), a DigitalVideo Disk (DVD), a digital tape, read/write (R/W) CDs, R/W DVDs, etc.

The one or more processors 152 of controller 150 may be configured toexecute instructions stored in the memory 154 so as to carry out variousoperations and method steps/blocks described herein. The instructionsmay be stored in a permanent or transitory manner in the memory 154.

FIG. 2 illustrates an operating scenario 200 of the system 100,according to example embodiments. While the operating scenario 200illustrates certain operations or blocks being in a certain order andbeing carried out by certain elements of system 100, it will beunderstood that other functions, orders of operations, and/or timingarrangements are contemplated herein.

Block 210 may include the controller 150 causing the at least one lightsource 130 to illuminate at least a portion of the scene withillumination light according to a structured light pattern. Thestructured light pattern could include, for example, at least one of: apredetermined light pulse repetition rate, a predetermined light pulseduration, a predetermined light pulse intensity, or a predeterminedlight pulse duty cycle.

In some embodiments, the structured light pattern could remain staticover one or more scans within a given field of view. Alternatively oradditionally, the structured light pattern could change dynamically. Forexample, the structured light pattern could be adjusted based on objectswithin the environment, a region of interest within the field of view; atime of day, presence of retroreflectors, etc. In some embodiments, thestructured light pattern could include a checkerboard pattern, a specklepattern, or a striped pattern.

In some example embodiments, in response to determining a retroreflectorwithin a portion of a given field of view, the intensity of the sectorthat had illuminated the retroreflector could be “dialed down” (e.g.,reducing a preamplifier gain or otherwise changing how a photosignalfrom that sector is processed in the analog and/or digital domain)and/or completely turned off or ignored to avoid blooming effects. Insuch a manner, the sensor may be better able to recover the remainingportions of the scene.

Periodically (e.g., once every few of frames within a maximum latency),the portion of field of view with the retroreflector could beilluminated again to track the presence of the retroreflector. If thesensor continues to indicate strongly saturated pixels in response toillumination (e.g., indicating that the retroreflective object is stillpresent in that region of the field of view), high energy illuminationwill not be provided to the given region until such time that the systemno longer observes a retroreflector in that region. Such dynamicillumination could reduce or eliminate stray light from retroreflectorsand more reliably recover the remainder of the scene which mightotherwise not produce reliable depth values. Without limitation, othertypes of spatial, temporal, and/or spectral light patterns arecontemplated herein.

In an example embodiment, instruction 212 could include, for example, asignal from the controller 150 to the light source 130 at time to. Theinstruction 212 could be indicative of the desired structured lightpattern and/or an illumination schedule, an illumination level, or anillumination direction or sector, among other examples.

In response to receiving the instruction 212, the light source 130 couldcarry out block 214 to illuminate the scene according to the structuredlight pattern. In some examples, the light source 130 could illuminateone or more light-emitter elements, which could be light-emitting diodes(LEDs), lasers, strobe lights, or another type of light source. Suchlight-emitter elements could be illuminated so as to provide the desiredstructured light pattern (e.g., provide light along a desired set ofpointing/cone angles, illuminate light-emitter elements for a desiredtime, illuminate light-emitter elements at a desired frequency and dutycycle, etc.). In some embodiments, the light source 130 could include anoptical element, such as one or more lenses, and/or a baffle so as todirect light toward a desired set of pointing angles and/or cone angle.

Block 220 includes causing the at least one ToF sensor 110 to provideinformation (e.g., time of flight information) indicative of a depth mapof the scene based on the structured light pattern provided by the lightsource 130. For example, at time t₁, block 220 could include providingan instruction 222 from the controller 150 to the ToF sensor 110. Theinstruction 222 could include a signal to trigger a depth mappingfunction of the ToF sensor 110. Additionally or alternatively, theinstruction 222 could include information indicative of a desired fieldof view for scanning, a desired range for scanning, a desiredresolution, and/or other desired aspects of the depth map and/or ToFsensor scan.

Block 224 could include the ToF sensor 110 obtaining a depth map based,at least in part, on the structured light pattern provided by the lightsource 130. That is, in response to receiving the instruction 222, theToF sensor 110 may carry out a depth-mapping scan of a field of view ofa scene. In an example embodiment, the ToF sensor 110 could be operatedbetween 10-100 MHz, however other operating frequencies are possible. Insome embodiments, the operating frequency of the ToF sensor 110 may beadjusted based on a desired maximum depth sensing range. For instance,the ToF sensor 110 could be operated at 20 MHz for a desired depthsensing range of approximately 7.5 meters. In some embodiments, the ToFsensor 110 could have a maximum desired depth sensing range of 100meters or more. In some embodiments that involve multiple ToF sensors,the ToF sensors could be configured to and/or instructed to carry outdepth-mapping scans of different fields of view of the scene and/or overdifferent distance ranges.

At time t₂, upon obtaining the depth map according to block 224, the ToFsensor 110 could provide information 226 to the controller 150. Theinformation 226 may be indicative of the depth map of the scene. Forexample, the information 226 could include a distance-based point map ofthe scene. Additionally or alternatively, the information 226 couldinclude a surface map of objects determined within the scene. Othertypes of information 226 are possible and contemplated.

Block 230 includes causing the imaging sensor 120 to provide informationindicative of an image of the scene based on the structured lightpattern provided by the light source 130. As an example, at time t₃, thecontroller 150 could provide an instruction 232 to the imaging sensor120. The instruction 232 could include a signal for triggering an imagecapture function of the imaging sensor 120. Furthermore, the instruction232 could include information regarding a desired exposure, ambientlighting level, ambient lighting color temperature, time of day, etc.While t₁ and t₃ are illustrated in FIG. 2 as being different, in someembodiments, times t₁ and t₃ could be similar or identical. That is, insome embodiments, at least some portions of the depth mapping and imagecapture processes could be triggered and conducted in parallel.

Block 234 includes, in response to receiving the instruction 232, theimaging sensor 120 obtaining an image of the scene illuminated by thestructured light pattern. In other words, instruction 232 could triggera physical shutter mechanism or a digital shutter so as to initiate animage capture process.

Upon capturing the image, at time t₄, the image sensor 120 could provideinformation 236 to the controller 150. The information 236 couldinclude, for example, the captured image as well as other information,such as metadata regarding the captured image (e.g., exposure time,aperture setting, imager sensitivity (ISO), field of view extents,etc.). In some embodiments, the information 236 could include RAW imagedata, however other uncompressed and compressed image data formats (BMP,JPEG, GIF, PNG, TIFF, etc.) are possible and contemplated.

Block 240 could include determining a high-resolution depth map of thescene based on the depth map of the scene (e.g., information 226) andthe image of the scene (e.g., information 236). In an exampleembodiment, the depth map information 226 and the image information 236could be compared and/or correlated using various image processingalgorithms. Such algorithms may include, without limitation, texturesynthesis, image resampling algorithms, interpolation algorithms, imagesharpening algorithms, edge-detection algorithms, and image blurringalgorithms, etc. As such, the high-resolution depth map could includedepth information about the scene with a higher spatial resolution thanthat of the depth map obtained by the ToF sensor 110. In someembodiments, the spatial resolution could relate to a target resolutionat a given distance away from the system 100. Other spatial resolutions,both along a two-dimensional surface and within three-dimensional space,are possible and contemplated herein. As an example, the depth mapobtained by the ToF sensor 110 could provide a spatial resolutionbetween adjacent sampling points of 10 centimeters at a range of 20meters. The high-resolution depth map could provide a spatial resolutionof less than 5 centimeters at a range of 20 meters. In otherembodiments, a high-resolution depth map could include other spatialresolutions that may be sufficient to sense objects (e.g., othervehicles, pedestrians, obstacles, signs, signals, etc.) within a fieldof view of the system 100.

Block 250 may include determining at least one inference about the scenebased on the depth map of the scene and, optionally, the image of thescene. For example, the controller 150 could determine at least oneinference about the scene based on the high-resolution depth mapdetermined in block 240. In such a scenario, the at least one inferencemay include information about one or more objects in an environment of avehicle or an operating context of the vehicle. In scenarios where thecontroller 150 includes a deep neural network, block 250 could beperformed, at least in part, by the deep neural network.

While the operating scenario 200 describes various operations or blocks210, 220, 230, 240, and 250 as being carried out by the controller 150,it will be understood that at least some of the operations of operatingscenario 200 could be executed by one or more other computing devices.

While operating scenario 200 describes various operations, it will beunderstood that more or fewer operations are contemplated. For example,the operations could further include selecting an illumination schedulefrom among a plurality of possible illumination schedules so as toprovide a desired exposure for the imaging sensor 120.

FIGS. 3A, 3B, and 3C illustrate various embodiments of the system 100and its elements. FIG. 3A illustrates a vehicle 300, according to anexample embodiment. The vehicle 300 may include one or more sensorsystems 302, 304, 306, 308, 310, 354 a-d, and 356 a-d. In some examples,the one or more sensor systems 302, 304, 306, 308, and 310 could includeLIDAR and/or radar sensor units. One or more of the sensor systems 302,304, 306, 308, and 310 may be configured to rotate about an axis (e.g.,the z-axis) perpendicular to the given plane so as to illuminate anenvironment around the vehicle 300 with light pulses and/or radarenergy. Additionally or alternatively, one or more of the sensor systems302, 304, 306, 308, and 310 could include a movable mirror so as todirect emitted light pulses and/or radar energy in the environment ofthe vehicle 300. For LIDAR-based sensors, determining various aspects ofreflected light pulses (e.g., the elapsed time of flight, polarization,etc.,) may provide information about the environment as describedherein. Similarly, radar-based sensors may determine information about agiven scene based on how radar energy interacts with the environment.

In an example embodiment, sensor systems 302, 304, 306, 308, and 310 maybe configured to provide respective point cloud information or othertypes of information (e.g., maps, object databases, etc.) that mayrelate to physical objects within the environment of the vehicle 300.While vehicle 300 and sensor systems 302 and 304 are illustrated asincluding certain features, it will be understood that other types ofsensors are contemplated within the scope of the present disclosure.

FIG. 3B illustrates a front view of sensor unit 350, according to anexample embodiment. Sensor unit 350 could include a housing 352. In someembodiments, the housing 352 could be coupled to, or integrated into,the vehicle 300. In an example embodiment, the sensor unit 350 mayoptionally include an imaging sensor 354, which could be similar oridentical to imaging sensor 120, as illustrated and described inreference to FIG. 1. Additionally, the sensor unit 350 could include aToF sensor 356, which could be similar or identical to ToF sensor 110,as illustrated and described in reference to FIG. 1. While FIG. 3Billustrates imaging sensor 354 and ToF sensor 356 as being disposedwithin a common housing 352, the imaging sensor 354 and ToF sensor 356could be disposed in different locations. It will be understood thatother arrangements of such elements are possible and contemplatedherein.

FIG. 3C illustrates a light source 370, according to an exampleembodiment. Light source 370 could include a housing 372. In someembodiments, the housing 372 could be coupled to, or integrated into,the vehicle 300. In an example embodiment, the light source 370 mayinclude a plurality of light-emitting elements 374 a-h, which could besimilar or identical to light source 130, as illustrated and describedin reference to FIG. 1. Light-emitting elements 374 a-h could bedisposed in an array or in another spatial arrangement. In an exampleembodiment, the light-emitting elements 374 a-h could be light-emittingdiodes (LEDs) or laser diodes. Other types of light sources are possibleand contemplated.

The light-emitting elements 374 a-h could be configured to emit light inthe infrared (e.g., near infrared 700-1050 nm) wavelength range.However, in some embodiments, other wavelengths of light arecontemplated (e.g., 1550 nm). In some embodiments, the light-emittingelements 374 a-h could be configured to emit light at differentwavelengths from each other. That is, the light-emitting elements 374a-h could be configured to emit light at eight different wavelengths. Insuch scenarios, system 100 and/or vehicle 300 could be configured todisambiguate light signals emitted by discrete light-emitting elements(or between different light sources 370) based on its wavelength. Insome embodiments, the multi-color light could be received by multi-colorimaging sensors and/or multi-color ToF sensors.

In some embodiments, light-emitting elements 374 a-h could include oneor more optical elements configured to interact with the light emittedfrom the light-emitting elements 374 a-h. Without limitation, the one ormore optical elements could be configured to redirect, shape, attenuate,amplify, or otherwise adjust the emitted light. For example, the one ormore optical elements could include a mirror, an optical fiber, adiffractive optic element, an aspherical lens, a cylindrical lens, or aspherical lens. Other types of optical elements are possible andcontemplated.

In some example embodiments, the light-emitting elements 374 a-h couldbe operable so as to emit light toward different spatial sectors (e.g.,including different azimuthal angle ranges and/or elevation angleranges) of the environment around vehicle 300. Furthermore, in someembodiments, the light-emitting elements 374 a-h could be operable toemit light at different times during a given period of time. That is,each of the light-emitting elements 374 a-h could be controlled to emitlight during respective time periods over a given time span. Forexample, the light-emitting elements 374 a-h could emit light in aserial pattern (e.g., one light-emitting element lit after another in a“chase” pattern). Additionally or alternatively, one or more of thelight-emitting elements 374 a-h could emit light in a parallel fashion(e.g., several light-emitting element emitting light simultaneously).

Returning to FIG. 3A, vehicle 300 could include a plurality of sensorunits, which could be similar or identical to sensor unit 350, asillustrated and described in reference to FIG. 3B. Furthermore, therespective sensor units could each include imaging sensors 354 a-d andToF sensors 356 a-d. As illustrated, the respective pairs of imagingsensors 354 a-d and ToF sensors 356 a-d could be coupled to, orintegrated into, a front, right side, left side, and rear portion of thevehicle 300. Other mounting types and mounting locations arecontemplated for the imaging sensors 354 a-d and ToF sensors 356 a-d.For example, in some embodiments, the imaging sensors 354 a-d and ToFsensors 356 a-d could be disposed in a rotatable mount configured torotate about the z-axis so as to obtain imaging information and ToFinformation from an environment around the vehicle 300.

While sensor systems 354 a/356 a, 354 b/356 b, 354 c/356 c, and 354d/356 d are illustrated as being collocated, it will be understood thatother sensor arrangements are possible and contemplated. Furthermore,while certain locations and numbers of sensor systems are illustrated inFIGS. 3A-3C, it will be understood that different mounting locationsand/or different numbers of the various sensor systems are contemplated.

Vehicle 300 could include a plurality of light sources 370 a-d, whichcould be similar or identical to light source 130, as illustrated anddescribed in reference to FIG. 1. As illustrated, light source 370 a-dcould be coupled to, or integrated into, a front, right side, left side,and rear portion of the vehicle 300. Other mounting types and mountinglocations are contemplated for the plurality of light sources 370 a-d.For example, in some embodiments, the light source 370 could be disposedin a rotatable mount configured to rotate about the z-axis so as to emitlight toward a controllable azimuthal angle range.

FIG. 4A-4B illustrate various sensing scenarios 400 and 420. In eachcase, for purposes of clarity, the sensing scenarios 400 and 420 mayillustrate a subset of possible spatial sectors and sensorprofiles/ranges. It will be understood that other spatial sectors arepossible and contemplated within the scope of the present disclosure.Furthermore, it will be understood that the sensing scenarios 400 and420 may illustrate only single “snapshots” in time and that spatialsectors and sensor profiles/ranges could be dynamically adjusted so asto periodically or continuously change based on, among other factors, adynamically-changing operating context of the vehicle 300.

FIG. 4A illustrates an overhead/top view of vehicle 300 in a sensingscenario 400, according to an example embodiment. Sensing scenario 400includes illuminating a front-facing sector of an environment of thevehicle 300 with structured light pattern 402. For example, light source370 a could emit light from one or more light-emitting elements so as toilluminate the front-facing sector of the vehicle 300 with thestructured light pattern 402.

The structured light pattern 402 could be provided according to a pulsedillumination schedule or a continuous-wave illumination schedule. Othertypes of illumination schedules are contemplated. For example, thestructured light pattern 402 could be provided “on-demand” fromcontroller 150 or based on the operating context of the vehicle 300. Asan example, the structured light pattern 402 could be provided inlow-light conditions (e.g., at night) or in response to determining anobject in the environment of the vehicle 300. As a non-limiting example,another sensor system of the vehicle 300 could identify an ambiguous orunknown object (not illustrated) ahead of the vehicle 300. The ambiguousor unknown object could be identified for further analysis. In such ascenario, the controller 150 could cause the light source 370 a toprovide the structured light pattern 402 to the front-facing sector.

While FIG. 4A illustrates a front-facing sector as being illuminated, insome embodiments, the light source 370 a may be configured to adjust apointing direction of the structured light pattern 402. It will also beunderstood that the other light sources 370 b-d could provide similarstructured light patterns into various spatial sectors correspondingwith their respective positions. For example, light source 370 d couldemit light according to the structured light pattern into a rear-facingspatial sector.

It will be understood that while the structured light pattern 402 andspatial sectors appear as being two-dimensional in FIG. 4A-4B,three-dimensional spatial volumes are contemplated. For example, thestructured light pattern 402 and/or spatial sectors could be defined asbetween an azimuthal angle range and also between a maximum elevationangle and a minimum elevation angle.

FIG. 4B illustrates an overhead/top view of the vehicle 300 in a sensingscenario 420, according to an example embodiment. Sensing scenario 420could include imaging sensor 354 a obtaining light from a field of view404. At least a portion of the light obtained by the imaging sensor 354a could include reflected or refracted light after the structured lightpattern 402 interacts with the environment of the vehicle 300. The fieldof view 404 could include a front-facing spatial sector of the vehicle300. In some embodiments, the field of view 404 of the imaging sensor354 a could partially or fully overlap with the volume illuminated bythe structured light pattern 402. Based on the light obtained from fieldof view 404, the imaging sensor 354 a may provide an image of the scenebased, at least in part, on the structured light pattern 402.

Sensing scenario 420 also illustrates ToF sensor 356 a obtaining lightfrom a field of view 406. At least a portion of the light obtained bythe ToF sensor 356 a could be from structured light pattern 402 that hasinteracted with the environment of the vehicle 300. The field of view406 could include a front-facing spatial sector of the vehicle 300. Insome embodiments, the field of view 406 of the ToF sensor 356 a couldpartially or fully overlap with the volume illuminated by structuredlight pattern 402. Based on the light obtained from field of view 406,the ToF sensor 356 a may provide a depth map of the scene based, atleast in part, on the structured light pattern 402.

FIG. 4C illustrates various structured light patterns 430, according toexample embodiments. The various structured light patterns 430 couldinclude, for example, a vertical striped structured light pattern 432, adot array structured light pattern 434, a checkerboard structured lightpattern 436, a diagonal striped structured light pattern 438, a“dropout” structured light pattern 440, and/or a speckle structuredlight pattern 442.

FIG. 4D illustrates a structured light pattern 444, according to anexample embodiment. As an example, structured light pattern 444 couldinclude a horizontal striped structured light pattern 446. It will beunderstood that other structured light patterns are possible and each iscontemplated without limitation.

In some embodiments, an illumination level (e.g., brightness) of some orall portions of the structure light patterns 430 could be dynamicallyadjusted based on objects within the scene and/or prior informationabout the scene. As an example, the amount of illumination provided tovarious portions of the scene could be based on the presence ofpredicted or known highly-retroreflective objects. In a scenario, theToF sensor could capture an initial scan of the scene while illuminatingthe scene at a relatively low illumination level. As an example, theinitial scan could include a brief (e.g., 10 microsecond) illuminationperiod. Such an initial scan could provide information aboutretroreflectors present within the scene. A subsequent scan of the scenecould be performed at a relatively high illumination level (e.g., 100microsecond illumination period, or longer) for portions of the scenewhere the retroreflectors are not present. The subsequent scan couldinclude illuminating the portions of the scene having theretroreflectors at a relatively low illumination level to confirm thepresence of a highly reflective object.

For example, in reference to FIG. 4C, if a retroreflective region 435 ais identified within a given scene during an initial scan, thenillumination of that retroreflective region 435 a could be reduced withrespect to other regions 435 b of the scene during a subsequent scan. Bydynamically adjusting the illumination level within the scene, potentialblooming issues and/or other problems relating to retroreflectors couldbe avoided or reduced on a near-real-time basis. Other ways todifferentially illuminate certain portions of the scene with respect toother portions of the scene are contemplated and possible.

III. Example Methods

FIG. 5 illustrates a method 500, according to an example embodiment. Itwill be understood that the method 500 may include fewer or more stepsor blocks than those expressly illustrated or otherwise disclosedherein. Furthermore, respective steps or blocks of method 500 may beperformed in any order and each step or block may be performed one ormore times. In some embodiments, some or all of the blocks or steps ofmethod 500 may be carried out by elements of system 100. For example,some or all of method 500 could be carried out by controller 150, ToFsensor(s) 110, and/or imaging sensor(s) 120 as illustrated and describedin relation to FIG. 1. Furthermore, method 500 may be described, atleast in part, by the operating scenario 200, as illustrated in relationto FIG. 2. Yet further, method 500 may be carried out, at least in part,by vehicles 300 or 400 as illustrated and described in relation to FIG.3A, 4A, 4B, 6A, or 6B. Method 500 may be carried out in scenariossimilar or identical to scenario 400 as illustrated and described inrelation to FIGS. 4A, 4B, and 4C. It will be understood that otherscenarios are possible and contemplated within the context of thepresent disclosure.

Block 502 includes causing at least one light source to illuminate ascene with a structured light pattern. The structured light patterncould be similar or identical to structured light pattern 402, 432, 434,436, 438, 440, and 442, as illustrated and described in FIGS. 4A, 4B,and 4C. In example embodiments, the structured light pattern couldinclude at least one of: a temporal light pattern, a spatial lightpattern, a predetermined light pulse repetition rate, a predeterminedlight pulse duration, a predetermined light pulse intensity, or apredetermined light pulse duty cycle.

Block 504 includes receiving, from a time-of-flight (ToF) sensor,information (e.g., time of flight information) about the scene based onthe structured light pattern. In an example embodiment, the controller150 could cause the ToF sensor to initiate a depth scan based on thestructured light pattern. In some embodiments, a clock signal or triggersignal could be provided to the ToF sensor to synchronize it with theone or more light pulses emitted into the environment. Upon obtainingdepth map information, the ToF sensor could provide to the controller150 information indicative of the depth map to the controller 150 oranother element of the system 100.

Block 506 includes determining a depth map of the scene based on thereceived information. For example, determining the depth map of thescene could include calculating distances to objects in the environmentbased on the time of flight of light pulses emitted into theenvironment. Other ways to determine the depth map of the scene based onthe received information are contemplated.

Optionally, method 500 could include causing an imaging sensor toprovide information indicative of an image of the scene based on thestructured light pattern. In some embodiments, the controller 150 couldtrigger a mechanical or electronic shutter of the imaging sensor to openand obtain an image of the scene. Additionally or alternatively, thecontroller 150 could provide information about the scene (e.g., ambientlight level, specific sectors of concern, desired resolution, time ofday, etc.). Furthermore, the controller 150 or the light source 130could provide a clock signal or trigger signal so as to synchronize theimaging sensor and light source. Upon obtaining the image of the scene,the imaging sensor could provide information indicative of the image tothe controller 150 or another element of system 100.

Additionally or alternatively, method 500 could include selecting adesired structured light pattern from among a plurality of possiblestructured light patterns. In some embodiments, the desired structuredlight pattern could be selected so as to provide a desired exposure forthe imaging sensor. Additionally or alternatively, selecting the desiredstructured light pattern could be based on a number of variables,including external light level, other light sources, angle of sun, etc.As such, method 500 could include selecting and/or adjusting thestructured light pattern based on an amount of ambient light (e.g., asmeasured from an ambient light sensor), a time of day, and/or weathercondition.

Optionally, method 500 could include determining a high-resolution depthmap (e.g., a depth map with higher resolution than that provided by theToF sensor individually) of the scene based on the depth map of thescene and the image of the scene.

Block 508 includes determining at least one inference about the scenebased on the depth map of the scene and, optionally, the image of thescene. In some embodiments, the at least one inference could includeinformation about one or more objects in an environment of a vehicle oran operating context of the vehicle.

In example embodiments, determining the at least one inference could beperformed by at least one deep neural network. Additionally oralternatively, some or all blocks of method 500 could be carried out bycomputing systems implementing other types of artificialintelligence-based algorithms.

FIGS. 6A and 6B illustrate sensing scenarios in the context of thepresent disclosure. The sensing scenarios could relate to system 100(e.g., as illustrated and described in reference to FIG. 1), vehicle 300(e.g., as illustrated and described in reference to FIGS. 3A, 4A, and4B), and method 500 (e.g., as illustrated and described in reference toFIG. 5).

FIG. 6A illustrates a sensing scenario 600, according to an exampleembodiment. As illustrated in FIG. 6A, a vehicle 300 could be operatingin an environment that includes one or more objects. As shown, thevehicle 300 includes sensor units 302, 306, 308, and 310. For instance,the sensor unit 302 may include a first LIDAR (not shown) and a secondLIDAR (not shown). Further, for instance, each of the sensor units 306,308, and 310 may also include a LIDAR. As shown, the vehicle 300 mayadditionally include imaging sensors 354 a-d, ToF sensors 356 a-d andlight sources 370 a-d. It will be understood that the vehicle 300 couldinclude different numbers and/or arrangements of imaging sensors 354a-d, ToF sensors 356 a-d, and/or light sources 370 a-d.

As shown, the environment of the vehicle 300 includes various objectssuch as cars 614 and 616, road sign 618, tree 620, building 622, streetsign 624, pedestrian 626, dog 628, car 630, driveway 632, and lane linesincluding lane line 634. In some embodiments, these objects havedifferent reflectivities, which can make it more difficult to obtainaccurate depth map information. In accordance with the presentdisclosure, the vehicle 300 may perform the methods and processesherein, such as method 500, to facilitate autonomous operation of thevehicle 300 and/or accident avoidance by the vehicle 300.

FIG. 6B illustrates a sensing scenario 650, according to an exampleembodiment. In some embodiments, the vehicle 300 and its associatedlight sources could emit light into its environment according to one ormore structured light patterns 652 and 654. For example, as illustrated,a right-facing light source could illuminate the environment withstructured light pattern 654, which could include a checkerboardpattern. Furthermore, a front-facing light source could illuminate theenvironment with structured light pattern 652.

Other scenarios are possible as well. Thus, the present methods andsystems may facilitate autonomous operation and/or accidence avoidancefor a vehicle such as the vehicle 300 by utilizing one or more ToFsensors in combination with light sources that are configured toilluminate the environment with structured light patterns.

Systems and methods described herein may involve prior information aboutthe environment. Such prior information could include a high-fidelitythree-dimensional model of the local environment of a vehicle and/orwithin a scene of the ToF sensor. In such scenarios, the priorinformation could reside, at least in part, at the vehicle and/or at acentral or regional server.

In some embodiments, the prior information may be utilized incombination with the ToF information/depth map to better calibrate thesensors and/or to better localize the vehicle. That is, a comparisonbetween the prior information and at least one depth map could helpdetermine intrinsic and extrinsic characteristics of the ToF sensor. Insuch scenarios, the determined intrinsic and/or extrinsiccharacteristics could be used to calibrate the ToF sensor. Additionallyor alternatively, a comparison between the prior information and the atleast one depth map could include aligning or registering the priorinformation with the at least one depth map. In so doing, thealignment/registration process could help determine a more-accurateabsolute position, heading, speed, or other characteristics of thevehicle and/or other aspects of its environment. In other words, theprior information could be utilized in conjunction with the at leastdepth map to provide more accurate information about the vehicle thanthe sensor information taken alone. In such scenarios, the priorinformation could represent a reference frame within which the vehiclecould be localized.

FIG. 7 illustrates a method 700, according to an example embodiment.Blocks and/or elements of method 700 could be similar or identical tocorresponding elements of methods 500 or 600, as illustrated anddescribed in reference to FIGS. 5 and 6

Block 702 includes providing prior information, which includesthree-dimensional information of a scene. The prior information couldinclude, for example, image, ToF, and/or LIDAR data obtained previously.Prior information could additionally or alternatively include a map, apoint cloud, or depth map, or other types of information.

Block 704 includes causing at least one light source to illuminate thescene with a structured light pattern. The structured light patterncould be similar or identical to other structured light patternsdescribed herein.

Block 706 includes causing the at least one ToF sensor to provide timeof flight information indicative of a depth map of the scene based onthe structured light pattern. As described herein, the ToF sensor couldbe operated while illuminating the scene with the structured lightpattern. Doing so may provide more detailed information about the depthof objects in the scene.

Additionally or alternatively, the prior information could be utilizedto improve depth estimation. In such a scenario, the prior informationcould be projected into the depth map(s). Various methods (e.g., raytracing, Principle Components Ordination (PCoA), Non-metricMultidimensional Scaling (NMDS), or other methods) could be used toperform the projection of three-dimensional prior information onto thedepth map, each of which are contemplated herein. By projecting theprior information into the depth map, depth information coulddouble-checked, calibrated, verified, and/or estimated more accurately.

Yet further, the prior information could be utilized to performbackground subtraction. In such a scenario, the prior information couldinclude information about objects that are outside a relevant sensordepth (e.g., far away from the vehicle). In such situations, depth mapinformation corresponding to objects that are outside the relevantsensor depth could be ignored, discounted, deleted, and/or processed ata lower resolution than other, more relevant, regions of theenvironment.

Additionally, the prior information could be used, at least in part, todetermine where retroreflective objects may be within a givenenvironment. When a vehicle (and its ToF imaging system(s)) enter suchan environment, it can adjust operation of the system so as to mitigatethe effects of the retroreflective objects. For instance, the systemcould illuminate the environment corresponding to a knownretroreflective object at a lower intensity level as compared to otherregions of the environment. In such a scenario, the hybrid imagingsystem can avoid “blooming” or “blinding” effects that can occur due toretroreflective objects. Additionally or alternatively, the hybridimaging system may operate at a different modulation frequency and/orilluminate the illumination source at a different rate. Other ways tomitigate the effects of retroreflectors are possible and contemplatedherein.

In some embodiments, a plurality of frames/scans from the ToF sensorcould be utilized to obtain information about the scene, which could beutilized together with other information described in the presentdisclosure. For example, “optical flow” can be obtained by a pattern ofapparent motion of an object between two consecutive ToF frames. Theoptical flow could include, for example, a two-dimensional vector fieldthat includes the displacement of corresponding objects in the scenebetween a first ToF frame and a second ToF frame. Based on the opticalflow, distances to the objects can be inferred and/or predicted. Suchdistance information from the optical flow could be utilized toconstrain the range of depths estimated using ToF information. That is,the optical flow could provide further information about ranges ofobjects in a given scene. The rough depth information could be used todetermine operating parameters for the ToF sensor and/or theillumination source. Additionally or alternatively, the rough depthinformation could be used to bound or constrain a set of operatingparameters used by the system more generally.

The particular arrangements shown in the Figures should not be viewed aslimiting. It should be understood that other embodiments may includemore or less of each element shown in a given Figure. Further, some ofthe illustrated elements may be combined or omitted. Yet further, anillustrative embodiment may include elements that are not illustrated inthe Figures.

A step or block that represents a processing of information cancorrespond to circuitry that can be configured to perform the specificlogical functions of a herein-described method or technique.Alternatively or additionally, a step or block that represents aprocessing of information can correspond to a module, a segment, aphysical computer (e.g., a field programmable gate array (FPGA) orapplication-specific integrated circuit (ASIC)), or a portion of programcode (including related data). The program code can include one or moreinstructions executable by a processor for implementing specific logicalfunctions or actions in the method or technique. The program code and/orrelated data can be stored on any type of computer readable medium suchas a storage device including a disk, hard drive, or other storagemedium.

The computer readable medium can also include non-transitory computerreadable media such as computer-readable media that store data for shortperiods of time like register memory, processor cache, and random accessmemory (RAM). The computer readable media can also includenon-transitory computer readable media that store program code and/ordata for longer periods of time. Thus, the computer readable media mayinclude secondary or persistent long term storage, like read only memory(ROM), optical or magnetic disks, compact-disc read only memory(CD-ROM), for example. The computer readable media can also be any othervolatile or non-volatile storage systems. A computer readable medium canbe considered a computer readable storage medium, for example, or atangible storage device.

While various examples and embodiments have been disclosed, otherexamples and embodiments will be apparent to those skilled in the art.The various disclosed examples and embodiments are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

What is claimed is:
 1. A sensor system comprising: at least onetime-of-flight (ToF) sensor configured to receive light from a scene; atleast one light source configured to emit a structured light pattern;and a controller that carries out operations, the operations comprising:causing the at least one light source, over a first illumination period,to illuminate at least a portion of the scene with the structured lightpattern; identifying, based on information from the first illuminationperiod, one or more retroreflective regions in the scene; adjusting thestructured light pattern based on the one or more identifiedretroreflective regions; causing the at least one light source, over asecond illumination period, to illuminate the portion of the scene withthe adjusted structured light pattern; and causing the at least one ToFsensor to provide time of flight information indicative of a depth mapof the scene based on the adjusted structured light pattern.
 2. Thesensor system of claim 1, wherein the adjusted structured light patterncontains lower illumination levels for portions of the scene where theone or more retroreflector regions are present.
 3. The sensor system ofclaim 1, wherein the first illumination period is less than 10microseconds.
 4. The sensor system of claim 1, wherein the firstillumination period is shorter than the second illumination period. 5.The sensor system of claim 1, wherein the structured light patterncomprises at least one of: a predetermined light pulse repetition rate,a predetermined light pulse duration, a predetermined light pulseintensity, or a predetermined light pulse duty cycle.
 6. The sensorsystem of claim 1, wherein the at least one light source comprises atleast one of: a laser diode, a light-emitting diode, a plasma lightsource, a strobe light, a solid-state laser, or a fiber laser.
 7. Thesensor system of claim 1, wherein adjusting the structured light patterncomprises selecting a desired structured light pattern from among aplurality of possible structured light patterns, wherein causing the atleast one light source to illuminate at least a portion of the scenewith the adjusted structured light pattern comprises illuminating theportion of the scene according to the desired structured light pattern.8. The sensor system of claim 1, further comprising an imaging sensor,wherein the imaging sensor comprises a plurality of photosensitiveelements, wherein the plurality of photosensitive elements comprises atleast one million photosensitive elements, wherein the operationsfurther comprise causing the imaging sensor to provide informationindicative of an image of the scene based on the adjusted structuredlight pattern.
 9. The sensor system of claim 8, wherein the operationsfurther comprise determining a high-resolution depth map of the scenebased on the depth map of the scene and the image of the scene.
 10. Thesensor system of claim 8, wherein the at least one ToF sensor, theimaging sensor, and the at least one light source are coupled to acommon substrate.
 11. The sensor system of claim 1, wherein theoperations further comprise determining at least one inference about thescene based on the depth map of the scene.
 12. The sensor system ofclaim 11, wherein the at least one inference comprises information aboutobjects in an environment of a vehicle or an operating context of thevehicle.
 13. The sensor system of claim 11, wherein the controllercomprises at least one deep neural network, wherein the determining theat least one inference is performed by the at least one deep neuralnetwork.
 14. A system comprising: a plurality of sensor systemsconfigured to be coupled to a vehicle, wherein each sensor systemcomprises: at least one time-of-flight (ToF) sensor; at least oneimaging sensor, wherein the at least one ToF sensor and the at least oneimaging sensor are configured to receive light from a scene; at leastone light source configured to emit a structured light pattern; and acontroller that carries out operations, the operations comprising:causing the at least one light source, over a first illumination period,to illuminate at least a portion of the scene with the structured lightpattern; identifying, based on information from the first illuminationperiod, one or more retroreflective regions in the scene; adjusting thestructured light pattern based on the one or more identifiedretroreflective regions; causing the at least one light source, over asecond illumination period, to illuminate the portion of the scene withthe adjusted structured light pattern causing the at least one ToFsensor to provide time of flight information indicative of a depth mapof the scene based on the adjusted structured light pattern; and causingthe imaging sensor to provide information indicative of an image of thescene based on the adjusted structured light pattern.
 15. The system ofclaim 14, wherein the operations further comprise determining ahigh-resolution depth map of the scene based on the depth map of thescene and the image of the scene.
 16. The system of claim 14, wherein atleast one of the sensor systems comprises at least one ToF sensor and atleast one imaging sensor in a common housing.
 17. A method comprising:causing at least one light source, over a first illumination period, toilluminate a scene with a structured light pattern; identifying, basedon information from the first illumination period, one or moreretroreflective regions in the scene; adjusting the structured lightpattern based on the one or more identified retroreflective regions;causing the at least one light source, over a second illuminationperiod, to illuminate the scene with the adjusted structured lightpattern; receiving, from a time-of-flight (ToF) sensor, time of flightinformation about the scene based on the adjusted structured lightpattern; determining a depth map of the scene based on the receivedinformation; and determining at least one inference about the scenebased on the depth map of the scene.
 18. The method of claim 17, whereinthe at least one inference comprises information about objects in anenvironment of a vehicle or an operating context of the vehicle.
 19. Themethod of claim 17, wherein adjusting the structured light patterncomprises selecting a desired structured light pattern from among aplurality of possible structured light patterns, wherein causing the atleast one light source to illuminate the scene with the adjustedstructured light pattern comprises illuminating the scene according tothe desired structured light pattern.
 20. The method of claim 17,wherein adjusting the structured light pattern further comprisesadjusting the structured light pattern based on an amount of ambientlight or a time of day.
 21. A method comprising: providing priorinformation, wherein the prior information comprises three-dimensionalinformation of a scene; selecting a structured light pattern based onthe prior information; causing at least one light source to illuminatethe scene with the selected structured light pattern; and causing the atleast one ToF sensor to provide time of flight information indicative ofa depth map of the scene based on the selected structured light pattern.22. The method of claim 21, further comprising: comparing the priorinformation to the depth map of the scene; and based on the comparison,determine a localized position of a vehicle.
 23. The method of claim 21,further comprising: comparing the prior information to the depth map ofthe scene; and based on the comparison, determine a calibrationcondition of the ToF sensor.
 24. The method of claim 21, furthercomprising: projecting the prior information into or onto the depth mapof the scene; and based on the projection, determine a localizedposition of a vehicle.
 25. The method of claim 21, further comprising:determining a background portion of the prior information; andsubtracting or ignoring at least a portion of the depth map of the scenecorresponding to the background portion.
 26. The method of claim 21,further comprising: determining at least one retroreflective objectbased on the prior information; and while scanning a portion of thescene corresponding to the at least one retroreflective object,adjusting at least one operating parameter of the ToF sensor or the atleast one light source.